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Staff Report to the Board 



Staff reported at the October 1994 Board meeting on efforts by the Chancellor’s Office 
and campuses to implement the Oregon Assessment Model adopted by the Academic 
Council in fall 1993. This framework stipulates that, in addition to traditional assessments 
on a course-by-course basis, OSSHE undergraduates will be assessed to provide evidence 
about student performance at three critical transitions: admissions, midpoint, and 

graduation. This model allows OSSHE campuses to develop assessment activities 
appropriate to their institutional missions within the context of the model to provide 
quality assurance. OSSHE’s assessment initiative is designed to serve dual functions: (1) 
improve the lives of students, including their retention, learning, and success; and (2) 
demonstrate accountability to students and their parents, elected officials, other agencies, 
and employers. 

Despite recent and fast growth in state-mandated assessment of college students and 
graduates, OSSHE took a different approach largely because of the abysmal failure in 
other states of assessment approaches that used single externally mandated instruments. 
The OSSHE approach is consistent with the observations of Donald M. Stewart, president 
of The College Board in New York, that for assessments to be useful for improving higher 
education, they should be campus-based, faculty-developed, and mission-related 
{Trusteeship, March/April 1995). These qualities do not appear to describe the simple 
solution that some states demand. But they do describe the State System’s approach to 
assessment. OSSHE’s assessment activities are related to efforts to raise productivity in 
higher education — serving more students, improving student performance, and providing 
a more valued mix of services relative to costs. 

In the fall of 1994, incentive funds were given to each campus to support the 
implementation of the Oregon Assessment Model. Each campus developed a proposal 
to begin assessments related to the three checkpoints in the model not covered by their 
ongoing assessment efforts. The Chancellor’s Office targeted $200,000 for special 
projects in undergraduate student assessment. Most of these projects were completed in 
February 1995 and were used to develop an accountability report for the 1995 Legislative 
Session. Staff promised to share with the Board information about the outcomes of these 
projects at a later date. 

Although the undergraduate curriculum is designed to have accumulative effects on 
general education and major field learning, the focus of traditional assessment is on what 
students learn (theories, concepts, and skills) on a course-by-course basis. The faculty 
member evaluates the work of tlie student and assigns a grade based on the professor’s 
judgment about the quality of the work. This assessment results in a transcript of courses 
taken, their credit value, and earned grades. Upon completion of the published program 
requirements and standards, a degree is conferred. As evidence of productivity, colleges 
and universities report the number of degrees earned by students each academic year. 
OSSHE conferred 13,457 degrees in 1993-94. 
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By its nature, this traditional approach to assessment takes a narrow view of how students 
change and c,row in college. However, the results of participation in higher education are 
many and include changes in a number of dimensions including: general knowledge, 
major field knowledge, social functioning skills, professional/occupational skiLs, personal 
goals and aspirations, general attitudes, attitudes toward self and others, and relationships 
with employers, family, community, and society (Ewell, 1984). 

The assessment demonstration projects undertaken by OSSHE campuses embrace this 
broader perspective of higher education outcomes and rely on a variety of assessment 
approaches to capture the full range of student abilities. Campuses gathered evidence 
about the effects of higher education on student perfofmance in a number of dimensions. 

General Knowledge and Abilities. More than half of the campuses used 
performance-based examinations (some used commercially available standardized tests 
and others used faculty-developed tests) to measure critical thinking, communication, 
and problem-solving skills of undergraduates. These data will be used as a baseline 
to determine the value added to abilities given entering student characteristics. 

Learning Environment. Several campuses are monitoring instructional practices that 
are associated with increased student learning effects. Currently enrolled students End 
their classes supportive and engaging. Self-reported gains in skills are high. 

Major Field Knowledge. A number of approaches — standardized objective tests, 
portfolios, senior capstone experiences — are being used to assess major field 
knowledge, including the application of learning in real-world settings, by all OSSHE 
campuses. 

Degree Completion. About half of students who entered as a first-time freshman (in 
Fall 1987) at an OSSHE institution earned a bachelor’s degree at the institution they 
entered or another OSSHE institution within seven years or less. Other students left 
to complete two- or four-year degrees elsewhere. 

Professional Licensure in Selected Programs . OSSHE students who take 
professional licensure examinations tend to have pass rates substantially higher than 
national averages. 

Employment. More than 2,500 baccalaureate graduates were surveyed. At the end 
of the first year, 95 percent of OSSHE baccalaureate recipients were employed or 
continued their education, pnmarily in a graduate program. Of those employed, 80 
percent found jobs in Oregon. 
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Customer Satisfaction . More than 4,500 students, recent graduates, and alumni of 
OSSHE colleges and universities were surveyed this year. TTiey reported high levels 
of satisfaction with the contribution the collegiate experience made to their preparation 
for life. Surveys of employer satisfaction will be completed by the end of the year. 

The focus of assessment is on strengthening undergraduate education. These assessment 
activities are having an impact on OSSHE campuses. OSSHE colleges and universities 
are rethinking traditional ways of structuring collegiate learning environments and finding 
ways of actively involving students in their education to enhance student retention and 
learning. Using the experience gleaned from the assessment demonstration projects, the 
Academic Council is now considering the next steps for campus- and state-level 
assessment. OSSHE submitted a proposal to the Fund for tlie Improvement of 
Postsecondary Education to extend the development of the Oregon Assessment Model. 
If funds are received, this project would develop a viable statewide assessment system for 
postsecondary education that is sample-based, involves faculty at two- and four-year 
public institutions in Oregon in its development, allows parallel assessments at the System 
and campus levels, permits campus choice, and involves Oregon employers in discussions 
about knowledge and abilities college graduates need to be successful in the workplace. 

In sum, strategies for improving higher education productivity require mechanisms for 
gathering information from a variety of sources and on an ongoing basis. OSSHE’s 
assessment and programmatic efforts are critical to developing a supportive learning 
environment in which students persist, make changes, grow intellectually, earn degrees, 
and bring their talents into Oregon’s workforce. Assessment plays an essential role in 
determining the success of our efforts and demonstrating accountability to our customers 
and investors. 



SUMMARY 

HIGHER EDUCATION: ASSESSMENT AND ACCOUNTABILITY 



Dimensions of student growth and change 

The results of participation in higher education 
are many and include changes in a number of 
dimensions including: 

• General knowledge 

• Knowledge of special fields 

• Social functioning skills 

« Professional/occupational skills 

• Personal goals and aspirations 

• General attitudes 

• Attitudes toward self and others 

• Relationships with employers, family, 
community, and society (Ewell, 1984). 

Traditional assessment of undergraduates 

For each course a student takes, the faculty 
member evaluates the work of the student and 
assigns a grade based on the professor’s judgment 
about the work’s quality. This assessment results 
in a transcript of courses taken, their credit value, 
and earned grades. Upon completion of the 
published program requirements and standards, a 
degree is conferred. As evidence of productivity, 
colleges and universities report the number of 
degrees earned by students each academic year. 

OS SHE conferred 13,457 degrees in 1993-94. 

Are there any shortcomings to this traditional 
approach? 

Although the undergraduate curriculum is 
designed to have accumulative effects on general 
education and major field learnings, the focus of 
traditional assessment is on what students learn 
(theories, concepts, and skills) on a course-by- 
course basis. 



OSSHE’s approach to quality assurance 

In Fall 1993 OSSHE adopted the Oregon 
Assessment Model. This framework stipulates 
that, in addition to course assessments, OSSHE 
undergraduates will be assessed to provide 
evidence about student performance at three 
critical transitions: admissions, midpoint, and 
graduation. 

A goal of the assessment model is quality 
assurance. Those who participate and invest in 
higher education should expect high duality. 

What purposes are served by the Oregon 
Assessment Model? 

OSSHE’s assessment initiative is designed to: 

• Improve the lives of students, including their 
retention, learning, and success; and 

• Demonstrate accountability to students and 
their parents, elected officials, other agencies, 
and employers. 

How is the Oregon Assessment Model 
consistent with OSSHE^s efforts to be more 
effective and efficient? 

OSSHE’s assessment activities are related to 
efforts to raise productivity in higher education — 
serving more students, improving student 
performance, and providir°: a more valued mix of 
services relative to costs. 

Strategies for improving higher education 
productivity require mechanisms for gathering 
information from a variety of sources and on an 
ongoing basis. 
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Assessment is critical to improving me quality of 
instruction, programs, advising, and student 
learning. The focus of assessment is on 
strengthening undergraduate education. 



How well are OSSHE students doing? 

In the fall of 1994, incentive funds were given to 
each campus to support the implementation of an 
assessment model. Campuses gathered evidence 
about the effects of higher education on student 
performance in a number of dimensions. 

Genera! Knowledge and Abi!itie& More than 
half of the cai.ipuses used performance-based 
examinations (some used commercially-available 
standardized tests and others used faculty- 
developed tests) to measure critical thinking, 
communication, and problem-solving skills of 
undergraduates. These data will be used as a 
baseline to determine the value added to abilities 
given entering student characteristics. 

Learning Environment. Several campuses are 
monitoring instructional practices that are 
associated with increased student learning effects. 
Currently enrolled students find their classes 
supportive and engaging. Self-reported gains in 
skills are high. 

Major Field Knowledge. A number of 
approaches - standardized objective tests, 
portfolios, senior capstone experiences — are 
being used to assess major field knowledge, 
including the application of learning in real-world 
settings, by all OSSHE campuses. 

Degree Completion. About half of students who 
entered as a first-time freshman (in Fall 1987) at 
an OSSHE institution earned a bachelor’s degree 
at the institution they entered or another OSSHE 
institution within seven years or less. Other 
students left to complete two- or four-year 
degrees elsewhere. 



Professional Licensure in Selected Programs. 
OSSHE students who take professional licensure 
examinations tend to have pass rates substantially 
higher titan national averages. 

Emp’':: ment More than 2,500 baccalaureate 
graduau ^ were surveyed. At the end of the first 
year, 95 percent of OSSHE baccalaureate 
recipients were employed or continued tlieir 
education, primarily in a graduate program. Of 
those employed, 80 percent found jobs in Oregon. 

Customer Satisfaction. More than 4,500 
students, recent graduates, and alumni of OSSHE 
colleges and universities were surveyed this year. 
They reported high levels of satisfaction with the 
contribution the collegiate experience made to 
their preparation for life. Surveys of employer 
satisfaction will be completed by the end of the 
year. 



What impact is assessment having on 
OSSHE campuses? 

OSSHE colleges and universities are rethinking 
traditional ways of structuring collegiate learning 
environments and finding ways of actively 
involving students in their education to enhance 
student retention and learning. OSSHE’s 
assessment and programmatic efforts are critical 
to developing a supportive learning environment 
in which students peisist, make changes, grow 
intellectually, earn degrees, and bring their talents 
into Oregon’s workforce. Assessment plays an 
essential role in determining the success of our 
efforts and demonstrating accountability to our 
customers and investors. 



TABLE OF CONTENTS 



Page 



Introduction 1 

General Knowledge and Abilities 3 

Learning Environment 5 

Major Field Knowledge 7 

Degree Completion: A Longitudinal View of First-Time Freshmen 9 

Professional Licensure in Selected Programs 11 

Employment 12 

Customer Satisfaction: Alumni 14 

Customer Satisfaction: Students 17 

Conclusion 19 



s 




ASSESSMENT AND ACCOUNTABILITY: 
INTRODUCTION 



How are students affected by the four-year 
college or university experience? 

The results of participation in higher education 
are many and include changes in a number of 
dimensions including: 

• General knowledge 

• Knowledge of special fields 

• Social functioning skills 

• Professional/occupational skills 

• Personal goals and aspirations 

• General attitudes 

• Attitudes toward self and others 

• Relationships with employers, family, 
community, and society (Ewell, 1984). 

What is the traditional way to assess 
undergraduate student learning? 

During a graduating senior’s typical years of 
collegiate study, he or she 

• Has spent 1,800 or more hours in the 
classroom, 

• Has written 75-100 essays, 

• Has taken 250 exams, and 

• Has completed numerous reports. 

For each course, the faculty member evaluates the 
work of the student and assigns a grade based on 
the professor’s judgment about the work’s quality. 
This assessment results in a transcript of courses 
taken, their credit value, and earned grades. Upon 
completion of the published program requirements 
and standards, a degree is conferred. Consistent 
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with this approach, colleges and universities 
report the number of degrees earned by students 
each academic year as evidence of productivity. 

Are there any shortcomings to this traditional 
approach? 

Although the undergraduate curriculum is 
design^ to have accumulative effects on general 
education and major field learnings, the focus of 
traditional assessment is on what students learn 
(theories, concepts, and skills) on a course-by- 
course basis. Like a photographer, newer 
assessment approaches use different lenses (or 
perspectives) to capture the abilities of students. 

How is OSSHE approaching quality 
assurance? 

In Fall 1993 OSSHE adopted the Oregon 
Assessment Model. This framew'ork stipulates 
that in addition to assessments associated with 
coursework, OSSHE undergraduates will be 
assessed at three critical transitions to provide 
evidence about student performance: admissions, 
midpoint, and graduation. 

This framework supports attention to inputs (e.g., 
characteristics of incoming students 
such as high school GPA or proficiency level) and 
outputs (e.g., desired attributes of graduates, 
including technical competence in a field, high- 
level intellectual abilities, employment). 

A goal of the assessment model is quality 
assurance. Those who participate and invest in 
higher education should expect high quality. The 
development of this quality assurance system 
involves assessing quality as well as 
communicating assessment results. 
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What is evidence of quality in higher 
education? 

The major types of evidence include: 

• The results of direct assessments of student 
abilities on graduation and the value added to 
abilities given entering student characteristics; 

• Performance in further education and on 
relevant licensure and certification 
examinations; 

• The successful and timely completion by 
students of their educational programs; 

• The placement and performance of graduates 
in the woriqplace and their effective 
involvement in civic and community life; and 

• Reported satisfaction of students with the 
contribution made by higher education 
toward the attainment of their goals 
{Education Commission of the States, 

1994 ). 

What are some specific ways campuses are 
pursuing assessment? 

OS SHE campuses were encouraged to apply 
existing rneth*^ Is when available, as well as 
develop their own approaches when appropriate, 
to gather mforraation about the set dimensions of 
student performance. 

In the fall of 1994, incentive funds were given to 
each campus to support the implementation of the 
assessment framework. Each campus identified 
activities consistent with the intention of the 
System’s approach to undergraduate assessment. 

Campuses used the Oregon Assessment Model as 
a basis to examine current assessment activities 
and to determine their area of greatest need. 



• Most campuses already evaluate the 
mathematics and writing abilities of entry- 
level freshmen for appropriate course 
placement. 

• The State System has been developing 
proficiency-based college admissions 
standards. 

• Because of these quality-control efforts now 
underway, campuses looked to assessing the 
effects of college programs on students at the 
mid and graduation checkpoints. 

What purposes are served by the Oregon 
Assessment Model? 

OSSHE’s assessment initiative is designed to 
serve dual functions of; 

• Improving the lives of students, including 
their retention, learning, and success; and 

• Demonstrating accountability to students and 
their parents, elected officials, other agencies, 
and employers. 

How is the Oregon Assessment Model 
consistent with OSSHE*s efforts to be more 
effective and efficient? 

OSSHE’s assessment activities are related to 
efforts to raise productivity in higher education ~ 
serving more students, improving instructional 
outcomes, and providing a more valued mix of 
services relative to costs. 

Strategies for improving higher education 
productivity require mechanisms for gathering 
information from a variety of sources and on an 
ongoing basis. 

Assessment is critical to improving the quality of 
instruction, programs, advising, and student 
learning. The focus of assessment is on 
strengthening undergraduate education. 
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GENERAL KNOWLEDGE AND ABILITIES 






General education curricula involve the basic 
study of liberal arts and sciences. The goal of 
general education is to foster the integration and 
synthesis of knowledge and the development of 
intellectual skills. These cognitive skills and 
habits of inquiry are associated with critical 
thinking, effective communication, and problem- 
solving. About one-third of a student’s 
baccalaureate program is devoted to general 
education courses. 



Abilities that last a lifetime 

Critical thinking, communication, and problem- 
solving abilities endure job and career changes 
throughout one’s working life. These abilities are 
vital for adult roles in a complex society. 

Educating critical thinkers 

Several campuses have been addressing academic 
concerns related to critical thinking, such as: 

What are the sldlls and dispositions that 
characterize critical thinking? What arc effective 
ways to teach critical thinking? How can these 
skills be measured? 

• Faculty at both Southern Oregon State 
College and Oregon Institute of Technology 
regularly meet to discuss how to introduce 
critical thinking into their courses. 

• Portland State University recently designed 
freshman courses (Freshman Inquiry) to 
improve critical thinking, communication, 
and other academic skills. These new 
courses are based on the examination of 
learning outcomes research and course-taking 
patterns of their students. 
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• Critical t*:inking is a required component of 
all of Oregon State University’s general 
education courses. 



Determining general education outcomes 

Because of the existing curricular emphasis on the 
development of critical thinking, several campuses 
examined ways to judge whether students are 
proficient in broad intellectual and analytical 
skills. Coordinators of assessment worked with 
faculty to develop an understanding about critical 
thinking and its assessment, select appropriate 
assessment tools, and determine conditions for 
assessment The campuses pursued several 
different assessment strategies. 

Southern Oregon State College used the 
Educational Testing Service (ETS) Tasks in 
Critical Thinking Test with a sample of 
freshmen and seniors. 

Oregon State University asked faculty 
teaching Baccalaureate Core courses, 
undergraduate students, and alumni who 
graduated within the last three years to rate the 
importance of critical thinking and whether 
OSU courses provide students with adequate 
training in critical thinking skills. 

Portland State University selected ACT’s 
College Outcomes Measurement Program 
(COMP) to assess skills needed for lifelong 
learning and success in employment. Work 
samples are also used to assess growth in 
knowledge and skills. 

Eastern Oregon State College evaluated the 
use of an existing Writing Proficiency 
Examination (developed by Eastern’s faculty) to 
assess student abilities in general 
communicative writing, advanced literacy, and 
critical thinking. 





Western Oregon State College uses student 
portfolios of writing samples to determine 
student abilities for fluent self-expression in 
English. Mathematics ability is assessed to 
determine student ability to use algebraic 
concepts in critical analysis and "real world" 
problem-solving. 

What did these assessments reveal about 

OSSHE students? 

Southern Oregon State College 

• About tw'O-thirds of seniors who sat for the 
exam are fully proficient in critical thinking 
(CT) tasks of inquiry, analysis, and 
communication. 

• When compared with the freshman sample, a 
greater proportion of seniors have attained 
superior performance in CT tasks. The 
largest difference in proficiency is seen in 
communication tasks. 

• A greater proportion of SOSC’s freshmen 
have "limited" or "no proficiency" in critical 
thinking tasks. 

Oregon State University 

• All 25 components of critical thinking were 
highly rated. Faculty emphasized a 
"reasoned discourse" model of critical 
thinking. Students and aliurmi viewed critical 
thinking as a set of attitudinal and social 
abilities which allow people to consider the 
views of others, to communicate their own 
views, and to negotiate differences in a non- 
confrontational manner. 

• Critical thinking skills were rated highly 
because they are viewed as basic to 
functioning in a diverse society which 
requires important decisions from its citizens. 

• Faculty, students, and alumni agree that OSU 
students receive adequate training in critical 
thinking skills. 



Eastern Oregon State College 

• Two-thirds of Eastern’s students pass the 
Writing Proficiency Examination (WPE) on 
t!iC first attempt. 

• The Critical Thinking Analytic Traits 
Assessment (CTAT) establishes a strong 
correlation between passing the WPE and 
applied critical thinking skills. 

• Eastern’s CTAT Rubrics effectively adapt the 
Oregon State Department of Education’s 
Dimensions of Reading Rubric to Eastern’s 
Writing Across the Curriculum Program, 
thereby providing a potentially powerful link 
between the state’s school reform objectives 
and Eastern’s curriculum. 

Western Oregon State College 

• Three-fourths of Western’s students 
demonstrate college-level writing proficiency; 
two-thirds demonstrate proficiency in 
mathematics. 

What Oregon employers say about critical 
thinking 

OSSHE is surveying Oregon employers to 
determine the critical thinking skills most 
important in the workplace and whether graduates 
of Oregon’s colleges and universities achieved 
these skills. These data will be available later this 
year. 

These efforts are important to developing baseline 
data about Objective 5 of the National Education 
Goals: "The proportion of college graduates who 
demonstrate an advanced ability to think critically, 
communicate effectively, and solve problems will 
increase substantially." 

More than half of the campuses used 
performance-based examinations (some used 
commercially-available standardized tests and 
others used faculty-developed tests) to measure 
critical thinking and communication skills of 
undergraduates. 




LEARNING ENVIRONMENT 



One approach to assessment of collegiate skills 
has centered on the development of performance- 
based examinations. However, given the high 
costs and long timelines for test development, the 
National Education Goals Panel recommends 
supplementing more direct cognitive assessments 
with indirect indicators of instructional practices 
that are associated with increased student learning 
effects. 

There are three broad categories of "good 
practice" indicators: 

• Institutional requirements that identify 
proficiencies or require specific experiences 
(such as a major research paper, math course, 
or foreign language course). 

• Instructional "good practices" used in 
undergraduate teaching (such as active 
learning, frequent feedback on performance, 
appropriate class sizes, peer interaction, 
writing requirements, out-of-class work 
requirements, and final examinations with 
essay or problem-solving components). 

• Student behaviors and self-reported gains in 
communication, critical thinking, or problem- 
solving skills. 

Research into collegiate teaching effectiveness has 
revealed, for example, that writing skills of 
undergraduates improve when students have 
opportunities to write. Likewise, participation in 
classroom discussions is linked to gains in critical 
thinking abilities. 

Indirect indicators of gain in higher order 
intellectual skills are found in student answers to 
questions, such as: Are writing assignments 
required in this class? Do students get timely 
feedback about their work? Do students 
participate in classroom discussions? 



A number of OS SHE campuses surveyed students 
and recent graduates to find out if the learning 
environments encountered in undergraduate 
classes are consistent with the literature on 
college teaching effectiveness. 

Portland State University. PSU selected the 
Classroom Environment Scales (Winston, et. al.) 
to measure quality and level of interaction 
between teacher and student, and among students, 
in the classroom, as well as opportunities for 
collaborative learning experiences. 

PSU students find their Freshman Inquiry classes 
supportive and engaging. 

• 92 percent of entering fiieshmen report they 
are in classes that are intellectually exciting. 

• 90 percent ux entering students intend to 
obtain a degree and prepare for a career. 

• 77 percent believe professors are willing to 
help outside of class. 

• 88 percent work together with other students 
on assignments and projects for the class. 

• 76 percent perceive that faculty encourage 
students to think for themselves. 

University of Oregon. UO conducted a 
telephone survey of students to obtain self-reports 
about learning experiences and outcomes. 

UO students arc actively engaged in learning. 

• 85 percent have seen an academic advisor. 

• 91 percent indicated they have discussed a 
course outside of class. 

• 83 percent have spoken with a faculty 
member outside of class. 

• 88 percent would recommend UO to a ftiend. 
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Percentage of undergraduates who think UO has 
improved their abilities in: 



Writing 


77% 


Gathering information 


91% 


Analyzing data 


91% 


Critical thinking 


93% 


Factual knowledge 


96% 


Understanding cultural differences 


80% 


Working independently 


84% 


Appreciating art 


77% 



OS SHE colleges and universities have made 
programmatic efforts in the last few years to 
assist entering students with their integration into 
the collegiate experience. The first year in 
college represents a very critical time of 
transition, especially the first quarter. 

In addition to the first term as an important 
transition, the teaching and learning environment 
in all of the classrooms and on the campus make 
a difference. The qualities of the environment 
that make a difference include: student-faculty 
contact, cooperation among students, active 
learning, prompt feedback, time on task, high 
expectations, and respect for diverse talents and 
ways of learning. These principles of good 
teaching and learning serve as a focus for 
strengthening undergraduate education. 



OSSHE students describe learning environments 
associated with increased learning outcomes. 
These include student experiences, faculty 
behaviors, and opportunities to extend learning 
beyond the classroom walls. Self-reported gains 
in knowledge and abilities acquired in college 
are high. 
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MAJOR FIELD KNOWLEDGE 



In addition to the generalized learnings associated 
with a baccalaureate, students select a major field 
in which to specialize. The selection of a major 
academic field is an important decision in the life 
of a student because the selection has an impact 
on career choice and development, future earning 
potential, and on an individual’s aesthetic, 
cultural, and intellectual attitudes. 

The more a student studies in a particular area of 
knowledge (for example, majoring in physical 
science, humanities, or engineering), the more 
he/she is expected to know related to the field. 
OSSHE institutions assess major field knowledge 
imd skills of seniors. Some use commerciaily- 
a^ailable major field tests, a few use portfolios of 
work samples, and others use senior projects or 
experiences for this purpose. 

What are the results of these major field 
assessments? 

Southern Oregon State College uses two 
approaches to major field assessment -- major 
field tests and capstone experiences. 

The major field tests developed by the 
Educational Testing Service (ETS) measure "the 
basic knowledge and understanding gained in the 
undergraduate curriculum ... the mastery of 
concepts, principles, and knowledge expected of 
students at the conclusion of a major in specific 
subject areas." SOSC used nationally-normed 
major field tests to assess student proficiency in 
biology, economics, and sociology. 

• Biology students ranked in the 79th 
percentile. 

• Sociology majors scored at the 82nd per- 
centile. Even higher scores were achieved by 
sfv-dents who took seven or more sociology 
courses. 



• Economics majors who took the test scored 
at the 51st percentile. 

Senior capstone projects were adopted at SOSC 
several years ago. These projects are intended to 
provide students with an integrative and in-depth 
experience in their field. 

• A sampling of capstone projects from the 
departments of Communication and Business 
was reviewed by employers of the larger 
Southern Oregon community to determine 
whether or not they met established goals. 

All reviev/ers concluded capstones met these 
goals and clearly reflected coursewoik and 
skills developed in the discipline. 

Oregon State University compared the use of 
two approaches for assessing knowledge of 
sociology majors — students took an objective test 
and developed portfolios of their work. Each 
portfolio was assessed by the instructor, the 
student, peers, and two teaching assistants at two 
points in the term. 

• The range of capabilities captured in the 
portfolio was greater than that suggested by 
objective test results. 

• An analysis of the standardized assessment 
scores revealed that students are competent 
judges of their own performances. 

• The single most accurate predictor of a 
student’s overall course performance was his 
or her final self-assessment. 

Oregon Institute of Technology requires senior 
students to complete projects demonstrating the 
application of knowledge to real problems. 

Under faculty supervision, students working alone 
or in teams review a broad variety of possible 
projects, research the topic, and determine the 
resources necessary to bring it to completion. 




During the process, students utilize the technical 
knowledge and skills gained during their 
education, as v/ell as skills in technical writing, 
speaking, and other allied topics. About 200 



proficiency. Lower performance for a subgroup 
of students may suggest the need for a department 
to examine courses, program requirements or 
instruction methods. 



senior projects are completed each year. 



using portable computers; a route congestion 
study; and a parking study. At the end of the 
term, these reports are given to tlie city and 
county for their use. 

• The low temperature heat engine. This 
project involves the development of an 
engine that is powered from 1 10-degree F. 
hot water, an idea patented by a Seattle 
inventor. The students have designed the 
system and are building the prototype. The 
eng, ne is a technological breakthrough using 
low temperature water heated by solar, 
geothermal, and waste heat sources to 
produce power that can be used to generate 
electricity or drive machinery. 

® The ground source heat pump. This project 
absorbs and rejects heat from the ground to 
heat and cool a residence. The technology 
has the potential of increasing from 20,000 
installed units per year presently to 400,000 
units per year by the year 2000, according to 
the Department of Energy. 



What is the value of major field assessment? 

Major field assessments are important tools fpr 
individual student advisement and program 
development Lower performance in a component 
of the major for an individual may suggest the 
need for additional coursework to raise 



• Traffic engineering technology. Student 



groups perfoina field traffic studies in 



Klamath Falls and surrounding areas. These 
include intersection turning movements and 
accident studies; intersection delay studies; 



OSSHE students are working in-depth in their 
fields, integrating knowledge of their discipline, 
applying knowledge to real-world contexts, 



traffic speed studies; a long-term traffic count 



evaluating their progress using portfolios, and 
performing well on standardized objective tests 
compared to national norms. 
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DEGREE COMPLETION: 

A LONGITUDINAL VIEW OF FIRST-TIME FRESHMEN 



More than half of the 1987-88 cohort of first-time 
freshmen graduated within seven years, either 
from the institution in which they first enrolled 
(44.4 percent) or from another (Dregon State 
System institution to which they transferred (6.7 
percent). Some who left will have completed a 
two- or four-year degree elsewhere, but there is 
no systematic way to track students once they 
leave the State System. 



Entering student characteristics 

Of the 8,142 full-time freshmen who entered an 
OS SHE institution in Fall 1987: 

• More than 80 percent were Oregon residents; 

• Nearly 10 percent were from a racial/ethnic 
minority group; and 

• Proportion of males and females was almost 
even. 

The timely and successful completion by 
students of their educational programs 
provides some evidence of quality. 

Of those first-time freshmen who entered in Fall 
1987: 

• 4,163 earned a four-year degree from an 
OS SHE institution. 

These data are comparable to national data for 
first-year attrition and degree completion for first- 
year freshmen.’ In addition to first-time 
freshmen, an estimated 6,000 students transferred 
to an OSSHE institution either from an Oregon 
community college or from a four-year college. 



Are some groups of students more successful 
than others? 

Attrition. Just over 26 percent of freshmen left 
the State System after the first year. 

• There is greater first-year attrition among 
men (28.1 percent) compared to women (24.5 
percent). 

• Among racial/ethnic groups, first-year 
attrition is lowest among Asian-American 
students (16.5 percent) and highest for 
American Indians (42.7 percent). 

• For residents and nonresidents, first-year 
attrition is about the same. 

Completion. Baccalaureate completion rates vary 
for different groups of students. 

Residency status. Residents are about as likely 
as nonresidents to graduate (51.5 percent versus 
49.5 percent). 

Gender. On average, females are more likely 
to earn a degree than males (53.5 percent 
versus 48.7 percent). 

Entering age. Students who delay entry to 
college are less likely to obtain their 
undergraduate degrees than are students who 
begin immediately after high school. 

• Fewer than 1 out of 3 students who began 
college in their 20s completed a degree at an 
OSSHE institution. 
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Race/ Ethnicity. Among first-time freshmen of 

different race/ethnicity: 

• Asian-American students (58 percent) are 
more likely to earn degrees than other 
racial/ethnic groups. 

• White/European American students (51.8 
percent) are more likely than 

either Hispanic (48.1 percent), African- 
American (30.2 percent), or American Indian 
(24.7 percent) students to earn a degree. 

Entrance Standards. Degree completion is 
strongly associated with individual ability. First- 
time freshmen of higher ability, as demonstrated 
by high school GPA, are more likely to earn a 
four-year degree from an OSSHE institution. 

• On average, almost 70 percent of first-time 
freshmen with high school GPAs of 3.25 or 
greater complete degrees. 

• The likelihood of earning a four-year college 
degree is more strongly associated with 
higher ability than racial/ethnic status. For 
students with high school GPAs of 3.25 or 
above, the percentage who completed degrees 
are: 



African-American 61.5% 

American Indian 41.0% 

Asian-American 74.5% 

Hispaiuc/Latino 67.5% 

White Euro. American 67.7% 



• Thirty-seven percent of first-time freshmen 
have a high school GPA of 3.25 or above: 



Afncan-American 


10% 


American Indian 


19% 


Asian-American 


44% 


Hispanic/Latino 


31% 


White Euro. American 


41% 


Total 


39% 



First-time freshmen with high school GPAs below 
2.5 have a much reduced chance of earning a 
degree (1 in 4). However, less than 10 percent of 
entering freshmen admitted have high school 
GPAs that low. 

How is assessment related to student 
retention and degree completion? 

Several campuses are collecting a range of 
student-related data — their intentions, attributes, 
perceptions, character of student exf>eriences, 
competing obligations of students, and the like. 
These assessments arc critical to developing a 
supportive learning environment in which students 
persist, make changes, grow intellectually, and 
earn degrees. Assessments of student goals, 
reasons for selecting a college, and support of 
family and peers are indicators of student 
commitment. Likewise, student self-assessments 
of knowledge, abilities and skills acquired arc 
important for determining if students are making 
the connections needed to be successful. These 
retention data provide a baseline against which to 
measure the effects of OSSHE’s efforts in the 
near future. 



One out of two first-time freshmen admitted in 
1987-88 to a State System institution earned a 
degree from either the institution in which they 
first enrolled or an institution in the State 
System. High school GPA — a combination of 
natural abilities, hard work and effort, and good 
study habits — is the best predictor of college 
success. 



* From Astin, Alexander W., "How Good is Your 
Institution’s Retention Rate?" Unpublished paper. 
University of California, Los Angeles, 1993; and Porter, 
Oscar F., Undergraduate Completion and Persistence at 
Four-Year Colleges and Universities. National Institute of 
Independent Colleges and Universities, 1989 
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PROFESSIONAL LICENSURE IN SELECTED PROGRAMS 



High quality professional education is essential to 
providing access to professional services for 
Oregonians. A large proportion of licensed 
professionals in their fields in Oregon attended an 
OSSHE institution. 

OSSHE offers entry professional programs in 
law, medicine, dentistry, and veterinary 
medicine. 

• 1,135 students were enrolled in entry 
professional programs in Fall 1994. 

• 300 professional degrees were awarded in 
1993-94 (116 in Law, 87 in Medicine, 69 in 
Dentistry, and 28 in Veterinary Medicine). 

In addition to meeting degree requirements, 
professional licensure or certification is required 
by state law for these professions as well as other 
fields. 



H OSSHE Pass Rale M National Pass Rate 




Medici ne Dentistry Law V eterinary 

Medicine 

OSSHE students who take these examinations 
tend to have pass-rates substantially higher than 
the national averages. 



Other fields require licensure examinations 
for graduates to practice in Oregon. 





OSSHE 


National 




Pass Rate 


Pass Rate 


Architectme 


85% 


56% 


Engineering * 


90% 


65% 


Accounting ** 


33% 


18% 


Pharmacy 


95% 


84% 



* OSU, PSU (1993) 

** PStJ, UO, OSU, EOSC, SOSC 



Students preparing for other occupations 
requiring licensure perform impressively on 
their exams as revealed in pass-rates. These 
include: 



Nuclear Medicine Technology 


100% 


Medical Technology 


90% 


Nursing 


90% 


Dental Hygiene 


100% 


Radiation Therapy 


100% 


Dietetic Internship 


100% 


Clinical Social Worker 


99% 


Registered Dietician 


94% 



The average income for Oregonians holding a 
professional degree is $67,600 (1992 survey by 
Employment Department). Households headed by 
professionals contribute a growing share of federal 
and state tax resources. 

Overall, OSSHE universities produce highly 
qualified professionals as evidenced by their 
exceptional petformance on national and state 
licensure examinations. 



NOTE: Educational professionals are required to take the 
National Test for Educators (NTE) beginning in 1994. 
Limited data are available this year. 
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EMPLOYMENT 



OSS HE contributes to Oregon* s pool of 
highly educated workforce needed for its 
future economic productivity. 

• Knowledge and abilities associated with a 
college education are valued in the 
workplace. 

» 39 percent of Oregon’s adults who hold 
baccalaureates graduated from an OS SHE 
institution (O.S. Census). 

Employment experiences of OSSHE 
graduates 

The effectiveness of OSSHE’s programs in 
preparing students is found in the employment 
experiences of students following graduation. 

OIT, OSU, and PSU surveyed recent graduates 
about employment experiences. 

Oregon Institute of Technology 

• 94 percent of OIT’s graduates in 1993-94 
were either employed, continuing their 
education, or not seeking employment 

• More than half found jobs in service 
industries (e.g., health care and engineering). 

• The average salary for 1993-94 OIT 
baccalaureate graduates was $30,312. The 
highest salary for a "fresh grad" was $52,000. 

• 75 percent of OIT graduates are employed in 
Oregon with 38 percent in the Portland area 
and 28 percent in Klamath Falls. 

• Since 1991, there has been a 21 percent 
increase of graduates finding work in Oregon. 



Oregon State University 

• 92 percent of OSU’s 1994-95 graduates are 
either employed, enrolled in graduate school, 
or not seeking employment. 

• 7 out of 10 are employed in a job related to 
their major. 

• 77 percent of OSU graduates are employed in 
Oregon, including one-third of graduates who 
were admitted as nonresident students. 

• Average salary of recent graduates employed 
full-time is $23,424. There are significant 
salary differences among majors. 

Portland State University 

• 98 percent are employed or enrolled in 
graduate school. 

• 64 percent are in jobs related to their majors. 

• 80 percent are employed in Oregon. 

• Almost two-thirds reported incomes in excess 
of $21,000. 

Employers rate skills and abilities of 
graduates 

OIT is working with American College Testing 
(ACT) to survey more than 400 employers who 
have hired OFT graduates. The instrument will 
measure employer satisfaction with the work 
habits, basic skills, technical/theoretical know- 
ledge, and personal qualities of OIT graduates. 
The survey will also gather information on 
emerging employer needs and recommenda-tions 
for improvement of the OIT curriculum. The 
survey will first be conducted in 1995. 
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Continuing education as indicator of 
graduate quality 

Following graduation, about 12 percent of recent 
graduates from OIT, OSU, and PSU are 
continuing their education; most arc enrolled in 
graduate school. 



OSSHE provides opportunities for Oregon* s sons 
and daughters to be full participants in the new 
economy. Of the 2JS00 baccalaureate graduates 
surveyed, 95 percent were employed or enrolled 
in graduate school. Of those employed, 80 
percent found jobs in Oregon. 



21 



o 



CUSTOMER SATISFACTION; ALUMNI 



The instruction component of public higher 
education has two broad categories of customers. 
First, those who directly consume higher 
education services — the students who attend and 
graduate (alumni) from the colleges and 
universities. Second, those who indirectly use 
higher education services through the employment 
of graduates. In order to serve our customers 
better, several institutions examined approaches to 
determine student, graduate, and/or employer 
satisfaction with the quality of education and 
services received on their campuses. 

What kind of job are we doing? 

Telephone interviews and mail surveys of alumni 
and recent graduates were professionally designed 
and conducted. 

Seven campuses siu^eyed more than 3,500 recent 
graduates or alunrni. They found very high levels 
of satisfaction with the contribution their 
institution made to students’ preparation for a 
fulfilling life. Almost all respondents believe 
college made a difference in their later success. 

More than 1,000 recent OSU graduates were 
surveyed. OSU students were satisfied with 
various aspects of the university and theu 
programs of study. 

• 9 out of 10 OSU graduates are satisfied with 
the quality of teaching and course content. 

• More than 8 out of 10 OSU graduates are 
satisfied with computing, laboratory, 
classroom, and library facilities. 

• More than 7 out of 10 OSU graduates are 
satisfied with course availability and size of 
classes. 
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• Communications skills courses in the OSU 
Baccalaureate Core are highly valued by 
these graduates. 

• The highest rated services provided by 
student support programs and offices are • 
study abroad, student activity center, cultural 
centers, and continuing higher education. 

• More than 4 out of 5 OSU students who used 
child care services, recreational facilities, 
student health services, and on-campus 
housing are satisfied. 

• When asked about the effects of OSU’s 
research mission on the quality of 
undergraduate instruction, recent graduates 
viewed the research mission as having 
improved the quality of their undergraduate 
education. 

Portland State University surveyed more 
than 1,000 students who graduated one year 
ago. 

• 78 percent were satisfied with service from 
faculty and staff. 

• 86 percent were satisfied with the quality of 
instruction. 

• 85 percent felt PSU prepared them for 
graduate school. 

• 89 percent were satisfied with their 
experience at PSU. 

• 54 percent gained practical employment 
experience while at PSU. 
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More than 1,000 University of Oregon alumni 
were interviewed by telephone. The sample 
included alumni who graduated from the UO 
between 1920 and 1994. 

• 8 out of 10 alumni said UO did a good job 
teaching undergraduates. 

• 8 out of 10 indicated faculty did a good job 
bringing their research into class. 

• 86 percent said they would select UO again. 

• 9 out of 10 indicated UO is important to 
attracting new business to Oregon. 

• The most frequently identified strength was 
"quality of UO’s education." 

• The most frequently identified weakness at 
UO was "state funding." 

More than 500 Western Oregon State College 
alumni were interviewed by telephone. 

• 9 out of 10 alumni said Western did a good 
job teaching undergraduates. 

• 93 percent said that "faculty cared." 

• Almost two- thirds completed a baccalaureate 
in 4 years or less. 

• 8 out of 10 alumni said location was 
important in original decision to attend. 

» 89 percent indicated they would select 
Western again. 

UO and WOSC alumni report great skill and 
knowledge gains in ability to: 



Write 


92% or more 


Think Critically 


96% 


Know Major Theories 


93% or more 


Public Speaking 


80% or more 


Exercise Leadership 


80% or more 



Oregon Health Sciences University surveyed 
dental and nursing graduates. 

Graduates of the dental school over the last 
six years were largely in solo practice 
earning a median annual income of 
$100,000 to $150,000. 

• 90 percent of the graduates are "positive" 
about preparation for general dental practice. 

• Two-thirds are positive about consulting with 
faculty about questions that arise in practice. 

Of the nursing graduates in 1991 and 1993, 

93 percent are currently employed, three-fourths 
full-time. 

• 17 percent have entered or completed a 
graduate program. 

• 7 out of 10 would recoinmend the program to 
people interested in a nursing care.er. 

• Graduates reported that the baccalaureate 
program achieved its learning goals. 

Oregon Institute of Technology has been 
working with American College Testing to survey 
alumni who graduated five and ten years ago to 
measure alumni continuing education, assessment 
of tlie value of their college experiences, and 
employment history. Alumni will be surveyed in 
1995. 

OSSHE graduates are very satisfied lifetime 
customers. 

The relationship isn’t over when the degree is 
awarded. Each graduate is viewed as a lifetime 
customer who brings family and friends to our 
campuses. 

• Alumni attend class reunions, lectures, 
performing arts events, and athletic events. 
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• They mentor current students, network with 
other alumni, and recruit new students. 

• More than 8 out of 10 alumni from UO and 
WOSC would recommend their alma mater 
to a high school student 

The information gleaned from customer 
perceptions enables campus administrators and 
faculty to see their work through the eyes of 
students, graduates, and employers and is valuable 
in achieving greater quality and productivity on 
the OSSHE campuses. 



Recent graduates and alumni report (hey are 
satisfied with the contribution their institutions 
made to their lives. 
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CUSTOMER SATISFACTION: STUDENTS 



Nothing is more central to the success of a 
business than meeting customer requirements. 

For higher education, students and employers are 
primary customers of the instructional component 
of higher education. A number of campuses 
surveyed currently enrolled students to obtain 
feedback about how satisfied they arc with the 
higher education services they rcceived ("the 
product") and the extent to which requirements 
are met 



Portland State University 

Portland State surveyed two groups of students -- 
entering freshmen and undergraduates. 

What Portland State freshmen say about their 
university: 

• 8 out of 10 are attending PSU as their first or 
second choice. 

• 8 out of 10 are likely to complete a 
bachelor’s degree at PSU. 

• 7 out of 10 are in classes they wanted. 

• 86 percent are in intellectually exciting 
classes. 

• 87 percent are socially comfortable coming to 
classes. 

Reasons freshmen arc attending PSU: 

• 88 percent to prcpare for a career. 

• 81 percent to enrich life. 

• 76 percent to increase income. 



What PSU undergraduates say about PSU: 

• 80 percent are employed outside the home. 

• 89 percent work more than 16 hours per 
week. 

• 79 percent are in classes they want to take. 

• 88 percent find classes intellectually 
stimulating. 

• 62 percent say instructors show concern for 
students. 

• 79 percent are satisfied with their major field 
instruction. 



University of Oregon 

What UO undergraduates say about their 
university: 

• 8 out of 10 say UO v/as their first or second 
choice. 

• 9 out of 10 are satisfied with the education 
they are receiving. 

• 9 out of 10 say UO does a good job teaching 
undergraduates. 

• 8 out of 10 say faculty' do a good job 
bringing research into classes. 

• 7 out of 10 have met a faculty member who 
cares. 

• 7 out of 10 think UO is academically 
challenging. 





17 



Reasons undergraduates are attending UO: 

• 8 out of 10 say faculty quality. 

• 8 out of 10 say location. 

• 7 out of 10 say prestige of degree. 



OSSHE students are satisfied with the higher 
education services they are receiving. 

Measuring customer satisfaction is an important 
part of our assessment effort. (The results of the 
employer surveys will be available later this 
year.) 



o 
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CONCLUSION 



Great expectations 

Those who participate and invest in higher 
education should expect high quality. The 
Oregon State System of Higher Education has 
adopted a number of initiatives related to 
increasing higher education productivity — serving 
more students, improving student performance, 
and providing a more valued mix of services 
rt-Iative to costs. 



Strategies for improving higher education 
productivity require mechanisms for gathering 
information from a variety of sources and on an 
ongoing basis. Assessment plays a key role in 
quality assurance. This document presented the 
baseline evidence collected this year to develop a 
quality assurance system. 



Students Beginning and Completing Degrees 
in State System Institutions 
1994 * 



New Admitted Students Degrees Completed 

Entering Fall 1 994” During 1 993-94 



New from Oregon 
high schools; 
•5,212 

New from 
high schools 
outside Oregon: 
•2,138 



> 



New fromOSSHE 
institutioi^: 

•693 Undergrad 
•390 Grad 

New from Oregon 
community colleges: 
•2,543 



New from Oregon 
independent colleges: 
•201 Undergrad 



• 75 Grad 



New from colleges 
outside of Oregon: 

• 1,643 Undergrad 

• 785 Grad 



All new from 
high school : 
•7,350 



New GED: 



•133 


All now 




Fall base enrcllment: 
•59,416 




All new from 


'w admitted 






'w 


colleges and 


^ students:** 




Includes: 


^ 


universities: 


•16,316 




•16,316 New 




•5,080 Undergrad 






•37,342 Continuing 




• 1,250 r -ad 






• 2,078 Nonadmitted New 

• 3,680 Retiuming after an 




New admits, 
source data not 






Absence 





on frie: 

•1,129 Undergrad 
•1,374 Grad 



All degrees 

awarded: 

•13,457 



Associates: 

•307 



Bachelors: 

•9,507 



Masters; 

•2,618 



Doctors: 

•418 



Professional 

•300 



Certificates: 

•244 



* Fall 1994 enrollment data and 1993-94 degrees data are used here to be consistent with data reported in the OSSHE Fact Book 1995. 
•* Excluding Oregon Health Sciences University for which data on new students are not available. 

Souces. (1) OSSHE Insfltuttonol Reseorch Sofvteei Poll 1994 Fourth Week Enfoflmeni reports, report ERTE-Ol . (2) IPEDS CompJetions reporli 1993-94 



^ I 





19 



How productive is OSSHE in transforming 
students into graduates? 

OSSHE’s base enrollment in 1994-95 is 59,416 
students. Of these students, 16,316 students 
began studies at an OSSHE college or university 
and 13,457 completed degrees. 

The newly admitted students include first-time 
freshmen, transfer students from Oregon’s 
community colleges, and transfer students from 
public and independent four-year colleges. 

This enrollment-to-degree production provides a 
panoramic view of the flow of students through 
the system and of OSSHE’s performance in one 
year. 

Degree production alone, however, does not 
describe the value added to students given their 
entering characteristics. In order to provide a 
closer look at what happens to students as a result 
of their participation in higher education, the 
Oregon Assessment Model was adopted as a 
framework to guide campus assessment efforts. 

Oregon Assessment Model 

OSSHE’s assessment approach, the Oregon 
Assessment Model, was adopted in the Fall 1993 
to improve student learning and to communicate 
more effectively with our customers and investors. 
In addition to assessment on a course-by-course 
basis, under tlie Oregon Assessment Model 
provides that undergraduates will be assessed at 
three critical transitions: as entering freshmen 
(admissions), for general knowledge and 
intellectual abilities (midpoint), and for major 
field knowledge and employment outcomes 
(graduation). 

Campuses are beginning to examine more closely 
the effects of what they do on student and 
graduate performance. 



• They aie using nationally normed tests, 
faculty developed tests, portfolio collections 
of student work, and self reports to assess 
gains in general knowledge and abilities and 
specialize knowledge in their major fields. 

• These campuses are using assessment to 
identify some common learning problems 
(such as reading, writing, speaking, 
quantitative reasoning, and critical thinking 
skills). The information gleaned from 
assessment will be used to improve the lives 
of students by strengthening the curriculum 
and targeting student advising to enhance 
learning effects. 

• In addition to performance-based assessment, 
campuses collected a range of student-related 
data - their intentions, attributes, perceptions, 
character of student experiences, and 
competing obligations of students. 

Assessment of student goals, reasons for 
selecting a college, and support of family and 
peers are indicators of student commitment 
and are related to persistence and degree 
completion. 

What effect is assessment having on OSSHE 
campuses? 

OSSHE colleges and universities are rethinking 
traditional ways of stmcturing collegiate learning 
environments and finding ways of actively 
involving students in their education to enhance 
student retention and learning. Assessment plays 
an essential role in determining the success of our 
efforts and demonstrating accountability to our 
customers and investors. OSSHE’s assessment 
and programmatic efforts are critical to 
developing a supportive learning environment in 
which students persist, make changes, grow 
intellectually, earn degrees, and bring their talents 
into Oregon’s workforce. 
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